Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Part-1 - LevelDbStore #3414

Open
wants to merge 36 commits into
base: master
Choose a base branch
from

Conversation

cschuchardt88
Copy link
Member

@cschuchardt88 cschuchardt88 commented Jul 8, 2024

Description

This fixes the store's dotnet memory management for GC, a long with adding new feature options, easier Exception catching.

Also exposes the store.
image

Change Log

  • Added more option features to ReadOptions, WriteOptions and Options.
  • Added LevelDBHandle.cs for GC and easy management.
  • Added IEnumerable<KeyValuePair<byte[], byte[]>> to Store.cs to Iterate over the whole store.
  • Changed namespace from Neo.IO.Data.LevelDB to Neo.IO.Storage.LevelDB
  • Added MaxOpenFiles=4096, SnappyCompression and FilterPolicy=10 option to Store.
  • Added code comments
  • Added thread-safety to batching since leveldb doesn't support multi-thread in batching.

Type of change

  • Optimization (the change is only an optimization)
  • Style (the change is only a code style for better maintenance or standard purpose)
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

Unit tests

Checklist:

  • My code follows the style guidelines of this project
  • I have performed a self-review of my code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • Any dependent changes have been merged and published in downstream modules

@cschuchardt88 cschuchardt88 added the Blocker Issues that are blocking other issues. Check issues details to see what it is blocking. label Jul 9, 2024
Copy link
Member

@vncoelho vncoelho left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can this PR be tested without PART 2?

@cschuchardt88
Copy link
Member Author

can this PR be tested without PART 2?

Yes

@Jim8y
Copy link
Contributor

Jim8y commented Jul 15, 2024

Hey Chris, can you please add a few UTs to demonstrate how your pr improves the leveldb? I am reviewing your code, but dont really clear how and where the improve works.

@cschuchardt88
Copy link
Member Author

cschuchardt88 commented Jul 16, 2024

Hey Chris, can you please add a few UTs to demonstrate how your pr improves the leveldb? I am reviewing your code, but dont really clear how and where the improve works.

Part-2 are the tests

@Jim8y
Copy link
Contributor

Jim8y commented Jul 16, 2024

Will check part 2 as well then, but please avoid doing this, cause it makes a single pr not reviewable.

@cschuchardt88
Copy link
Member Author

Will check part 2 as well then, but please avoid doing this, cause it makes a single pr not reviewable.

Just trying to make it small.

@cschuchardt88
Copy link
Member Author

Added thread-safety to batching since leveldb doesn't support multi-thread in batching.

@Jim8y
Copy link
Contributor

Jim8y commented Jul 30, 2024

@superboyiii can you please rake a look at this pr and do some tests on it?

@superboyiii
Copy link
Member

@superboyiii can you please rake a look at this pr and do some tests on it?

Sure

@superboyiii
Copy link
Member

@cschuchardt88 Seems not an issue from this PR because it occurs on master in block 2724677, I will open an issue.
1722997794683

Comment on lines +40 to +49
if (disposing)
{
FreeManagedObjects();
}
if (Handle != nint.Zero)
{
FreeUnManagedObjects();
Handle = nint.Zero;
}
_disposed = true;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I prefer to use Interlocked for thread safe flag use

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's reasonable I'll add it in when I get a chance

Copy link
Member Author

@cschuchardt88 cschuchardt88 Oct 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like you can't Interlocked. its a non-ref returning type. Reason for locking it?

@shargon
Copy link
Member

shargon commented Oct 11, 2024

@superboyiii Require testing in x86 and x64 machines

@cschuchardt88
Copy link
Member Author

@superboyiii Require testing in x86 and x64 machines

@shargon I added the nuints. I think I got them all.

@superboyiii
Copy link
Member

@superboyiii Require testing in x86 and x64 machines

OK

Copy link
Member

@superboyiii superboyiii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Retsted with Part2, it works well and stable. No data imcompatible. I don't have a way to get a x86 but it works well on x64.

@cschuchardt88
Copy link
Member Author

Retsted with Part2, it works well and stable. No data imcompatible. I don't have a way to get a x86 but it works well on x64.

@shargon any input would be nice. so we can get this going.

return db.Get(options, key);
}
public byte[] TryGet(byte[] key) =>
_db.Get(key, _readOptions);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that if we use lock to write we also need it to read, or you could be reading while someone else is writing

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's why you take/use a snapshot.

Native.leveldb_close(handle);
handle = IntPtr.Zero;
}
Put(key, value, new WriteOptions());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cache write options?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Blocker Issues that are blocking other issues. Check issues details to see what it is blocking. Plugins Tested Waiting for Review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants